One important note, I've noticed a couple of other things while making these changes:

• I named the config field worker, to be consistent with gpu. Should we update these to workers and gpus for clarity, to show we are specifying an amount? Or is it fine as is?
• I noticed that cluster.details() and our internal RayCluster object only show/hold one value for cpu, even though we actually have a min_cpu and max_cpu. Should these be updated to also show/hold both the min and the max properly? (aka the opposite problem of what we had with this PR lol)

It works try the following

cluster = Cluster(ClusterConfiguration(local_interactive=local_interactive, namespace=namespace, name=cluster_name, worker=1, min_cpus=1, max_cpus=1, min_memory=4, max_memory=4, gpu=0, instascale=False, machine_types=["m5.xlarge", "p3.8xlarge"]))

Looks good Mustafa, tried it with:

cluster = Cluster(ClusterConfiguration(
    name='mnisttest',
    namespace='default',
    worker=2,
    min_cpus=2,
    max_cpus=2,
    min_memory=2,
    max_memory=2,
    gpu=0,
    image="quay.io/project-codeflare/ray:2.5.0-py38-cu116",
    instascale=False, # Can be set to false if scaling not needed
    machine_types=["m5.xlarge", "g4dn.xlarge"] # Can be removed if above is false
))

I named the config field worker, to be consistent with gpu. Should we update these to workers and gpus for clarity, to show we are specifying an amount? Or is it fine as is?

I personally like num_workers and num_gpus if you are going to make a change. But definitely not soemthing worth blocking the PR on.

/LGTM Thanks Mustafa, I was able to run on my cluster. Still not quite certain how it all gets folded in so the num_workers and num_gpus begin to work with the codeflare_notebook, but LGTM in testing.

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jbusche, MichaelClifford

The full list of commands accepted by this bot can be found here.

The pull request process is described here